Hole Filling in Images

This project relates to automated 3D model generation of Urban Environments. Ground based modeling involves a setup of two 2D laser scanners, and a digital camera mounted on top of a truck. As we drive the truck in a city, the laser scans capture depth information using the LIDAR time-of-flight principle. These laser scans are then subjected to accurate localization and 3D data processing algorithms to create a 3D mesh of the urban environment. The resulting mesh is then texture mapped with camera images to produce photo-realistic models. In doing so, foreground objects such as trees, cars, lamposts etc. occlude parts of the background buildings from the laser scanners and the digital camera, and thus leave holes both in the geometry and texture. For a 3D model, ideally the user should be able to view the building façade that is occluded by a tree or other objects. In this report, we present a simple and efficient method for filling holes in images obtained through the above setup. Given an image with regions of unknown RGB values, our goal is to determine the missing pixel values from the remainder of the image. We use our proposed method to fill holes in the texture atlases generated during automated 3D modeling of Urban Environments. Our approach can also be used for other applications such as restoring old and damaged photographs, removing objects from images, special effects. Our method first fills low pixel variance regions by a pass of 1D horizontal interpolation in which for each row it linearly interpolates any missing values, if they lie in a low pixel variance region. This is followed by a pass of 1D vertical interpolation. A copy-paste method is then employed to synthesize the missing texture. A window is taken around the hole, and a matching region in the image is found. The hole is filled by copying the matching region and pasting it over the hole. The approach is found to work well on most images and does not suffer from the limitations of local inpainting in traditional hole filling schemes.